Data center architecture utilizing optical switches

ABSTRACT

Embodiments of the invention describe flexible (i.e., elastic) data center architectures capable of meeting exascale, while maintaining low latency and using reasonable sizes of electronic packet switches, through the use of optical circuit switches such as optical time, wavelength, waveband and space circuit switching technologies. This flexible architecture enables the reconfigurability of the interconnectivity of servers and storage devices within a data center to respond to the number, size, type and duration of the various applications being requested at any given point in time.

PRIORITY

This application is a continuation of U.S. application Ser. No.16/983,325, filed Aug. 3, 2020, which is a continuation of U.S.application Ser. No. 16/839,643, filed Apr. 3, 2020, which is acontinuation of U.S. application Ser. No. 16/588,031, filed Sep. 30,2019, which is a continuation of U.S. application Ser. No. 15/425,154,filed Feb. 6, 2017, which is a continuation of U.S. application Ser. No.14/191,208, filed Feb. 26, 2014, which claims the benefit of U.S.Provisional Application No. 61/770,244, filed Feb. 27, 2013, each ofwhich are hereby incorporated by reference herein in their entirety.

FIELD

Embodiments of the invention generally pertain to network computing andmore specifically to data center architectures utilizing optical circuitswitches.

BACKGROUND

Fueled by the increasing user demand for data such as images, video,multimedia and various databases, data centers are expected to grow fromtoday's petascale (i.e., computer systems capable of reachingperformance in excess of one petaflops, i.e. one quadrillion floatingpoint operations per second) to exascale (i.e., a thousand fold increaseover petascale). Moreover, the continued emergence of new, widelydiverse web services and cloud computing applications require futuredata centers to be more flexible/elastic. Another increasingly importantrequirement in future data centers is the ability to achievevery-low-latency, high-performance server-to-server connectivity, asemerging applications will likely be more complex and morecomputationally intensive, and thus will require much more interactionsamong servers compared to interactions with external clients.

BRIEF DESCRIPTION OF THE DRAWINGS

The following description includes discussion of figures havingillustrations given by way of example of implementations of embodimentsof the invention. The drawings should be understood by way of example,and not by way of limitation. As used herein, references to one or more“embodiments” are to be understood as describing a particular feature,structure, or characteristic included in at least one implementation ofthe invention. Thus, phrases such as “in one embodiment” or “in analternate embodiment” appearing herein describe various embodiments andimplementations of the invention, and do not necessarily all refer tothe same embodiment. However, they are also not necessarily mutuallyexclusive.

FIG. 1 is an illustration of a data center architecture utilizingmultiple levels of electronic packet switches according to an embodimentof the disclosure.

FIG. 2 includes illustrations of data center architectures utilizingoptical circuit switches according to embodiments of the disclosure.

FIG. 3 illustrates two-level folded clos switching architecturesaccording to embodiments of the disclosure.

FIG. 4 illustrates three-level folded clos switching architecturesaccording to embodiments of the disclosure.

FIG. 5A and FIG. 5B illustrates configurable bandwidths for switchingarchitectures according to embodiments of the disclosure.

Descriptions of certain details and implementations follow, including adescription of the figures, which may depict some or all of theembodiments described below, as well as discussing other potentialembodiments or implementations of the inventive concepts presentedherein. An overview of embodiments of the invention is provided below,followed by a more detailed description with reference to the drawings.

DESCRIPTION

Embodiments of the invention describe flexible (i.e., elastic) datacenter architectures capable of meeting exascale through the use ofoptical circuit switches such as optical time, wavelength, waveband andspace circuit switching technologies. This flexible architecture enablesthe ability to reconfigure the interconnectivity of servers and storagedevices within a data center to respond to the number, size, type andduration of the various applications being requested at any given pointin time.

Throughout this specification, several terms of art are used. Theseterms are to take on their ordinary meaning in the art from which theycome, unless specifically defined herein or the context of their usewould clearly suggest otherwise. In the following description numerousspecific details are set forth to provide a thorough understanding ofthe embodiments. One skilled in the relevant art will recognize,however, that the techniques described herein can be practiced withoutone or more of the specific details, or with other methods, components,materials, etc. In other instances, well-known structures, materials, oroperations are not shown or described in detail to avoid obscuringcertain aspects.

FIG. 1 is an illustration of a data center architecture utilizingmultiple levels of electronic packet switches according to an embodimentof the disclosure. Data center architecture 100 is based on a foldedClos interconnect topology (alternatively referred to as a “fat tree”topology) comprising level of servers 110 and multiple levels ofelectronic packet switches (e.g., ethernet switches). Clos networktopology provides constant or non-blocking bisectional bandwidth (i.e.,the bandwidth available at any level in the network) as the number ofinterconnected nodes (i.e., servers) increases, and also providesconstant latency for expanding numbers of nodes in the cluster. Closnetworks include “leaf” and “spine” levels. A leaf level is directlycoupled to a set of the input/output nodes, while a spine level switchessignals among the leaf stages, thereby enabling any leaf node tocommunicate with any other leaf node.

The illustrated data center architecture comprises four levels of packetswitches including leaf switches 120 (referred to herein as top-of-rack(ToR) (or edge) switches), two levels of aggregation/distributionswitches 130 and 140, and spine (or core) switches 150. Theaggregation/distribution switches may provide a redundant capability inthe event of switching circuit failure or unavailability, and mayfurther used to increase the bisectional bandwidth of the data center.The spine switching layer may further serve as a gateway to externalnetworks. Further operational details of folded clos data centerarchitectures are discussed below in the context of embodiments of theinvention described below.

FIG. 2 includes illustrations of data center architectures utilizingoptical circuit switches according to embodiments of the disclosure.Data center architecture 200 is illustrated as a folded Closinterconnect topology created using servers 210, top-of-rack electronicpacket switches 220 and spine electronic packet switches 250, similar todata center architecture 100 of FIG. 1 ; however, instead of theelectronic packet aggregation/distribution switches 130 and 140, aplurality of optical circuit switches for communicatively coupling theplurality of ToR switches to the plurality of spine switches is used. Inthis embodiment, wavelength division multiplexing (WDM) switches 230 andoptical time-division multiplexing (OTDM) switches 240 are used.

For data center architecture 100, all levels of switching utilizeelectronic packet switching devices; packet switching features deliveryof packets (i.e., variable-bit-rate data streams) over a shared network.When traversing the switching hardware of levels 120-150, packets arebuffered and queued, resulting in variable latency and throughputdepending on the traffic load in the network.

In contrast, data center architectures according to embodiments of theinvention utilize optical circuit switching, which utilize dedicatedconnections of constant bit rate and constant delay between levels forexclusive use during the communication session. Optical circuitswitching enables a higher potential bit rate and lower loss compared toelectronic packet switching. Furthermore, optical circuit switchesconsume less power and enable the design of elastic data centers (i.e.,changeable data center architectures capable of dividing the data centerinto different configurations) as described below.

For data center architecture 200, optical circuit switch levels 230 and240 produce no buffering, delay or latency compared to levels 130 and140 of data center architecture 100. In this example, there exists threeelectronic switch delays total—data traversing switching network 200encounters ToR electronic packet switch level 220, spine electronicpacket switch level 250, and ToR electronic packet switch level 220again. In contrast, data center architecture 100 contains sevenelectronic switch delays total (levels 120-140 twice, and spineelectronic packet switch level 150).

For folded clos interconnect data center architectures using purelyelectronic packet switching, all packet switches may have the samecapacity (2 N R), where N is the fan-out number per switching stage, andR is the bit rate per link—for a total of 2N³ racks, with M servers perrack, and a bit rate of S(=R N/M) per server. All links arebidirectional. Data storage devices may reside in the serversthemselves, or may be stand-alone devices replacing some of the servers.

In data center architecture 200, WDM switches 230 and OTDM switches 240may be deployed such that each link comprises N wavelengths, each ofwhich comprises N time slots. Each server may reach all of the N³ spineswitches (via N fibers×N wavelengths×N time slots).

In a non-blocking architecture, the bisectional bandwidth is the same(=2N⁴R) at all levels of the data center. FIG. 2 illustrates that eachof ToR switches 220 (and hence, each of servers 210), can reach all ofthe N³ spine switches (note that this implies that there are N³ distinctpaths between any pair of ToR switches). This results in a rich packetconnectivity among servers, which is highly desirable, but does notrequire special routing protocol needed for the data center architectureof FIG. 1 , including traffic engineering and load balancing.

Data center architecture 205 is illustrated to further include anadditional optical switching layer—ToR optical space (i.e., fiber)switch level 260, such that only one electronic switch delay, from spineelectronic packet switch level 250, exists. Thus, any layer of a purelyelectronic packet switching data center architecture may be replaced byan optical circuit switching layer. In some embodiments, at least onelevel of a data center architecture comprises an electronic packetswitching layer so that data transferred between the servers comprisespacket data. In some of these embodiments, said electronic packetswitching layer comprises either the ToR layer or the spine layer.

Thus, compared to data center architecture 200, as each ToR packetswitch is replaced with an optical spatial circuit switch in system 205each of servers 210 now has N fibers×N wavelengths×N time slots; thus itcan directly reach all of the N³ spine switches. The number ofpacket-switching layers is reduced to one, thereby reducing latency.Moreover, because the bisectional bandwidth (described in further detailbelow) is the same for all levels of the data center, and that the spinepacket switching layer is well connected to all servers (with N³distinct paths between any pair of servers), both the architectures 200and 205 are non-blocking architectures from a packet-switching point ofview.

To enable optical integration, the N wavelengths used in architectures200 and 205 may fit within a spectral region compatible with a chosendevice and circuit integration technology. In some embodiments, thisspectral region is limited to, for example, the C or C+L band. For N=16,for example, the wavelength separation may be 200 to 400 GHz. In someembodiment, thermoelectric coolers (TECs) are used to compensate forunavoidable thermal drifts across the data center; in other embodiments,athermal optical devices—i.e., devices having athermicity and a tunablespectral response for optical filters, are used in order to eliminatethe need for TECs. Various types of wavelength-selective components suchas WDM devices, asymmetric Mach-Zehnder interferometers (AMZIs) andarray waveguide gratings (AWGs) are commonly implemented in PICs.Athermicity of such components implies maintaining a consistentfrequency transmission spectrum as the ambient temperature changes. Atunable spectral response for these components describes changing theirspectrum based on their application, as well as potentially correctingfor fabrication deviations from the design.

In some embodiments of the invention, athermal operation of a filteringcomponent on a PIC is obtained via active control by heating a region ofthat component. A control feedback loop maintains a set point (i.e.,constant) temperature for the heated region; thus, in these embodimentsthere is no need to sense the ambient temperature or change the heatedregion's temperature as a function of the ambient temperature. Bychanging the set point temperature, the transmission spectrum of afiltering component may be linearly shifted to actively tune itsresponse.

Some embodiments may also utilize optical devices with design featuresfor improving the power efficiency of a device. In some embodiments, theabove described heated region for active control is thermally isolatedfrom the above described ambient region through modification of thethermal conduction path between the heated region and the ambientregion, such as through localized thermal engineering of the substrate.In some embodiments, devices are designed to have waveguide regionshaving different thermo-optic coefficients (alternatively referred toherein as ‘dn/dT,’ as described below), either from differing materialsor differing waveguide cross-sections, wherein a region where waveguideshave a high dn/dT is heated, while the remaining bulk of the filteringdevice uses waveguides with a low dn/dT. Control at the PIC scale can befurther simplified if an actively heated region is shared by two or moresuch devices.

Reducing N to 8 to double the wavelength separation does not produce ahigh enough radix to yield a large bisectional bandwidth—assuming thatall the switches, electronic and optical, have the same size. The Clostopology, in general, allows different size switches for the differentstages. Thus, a more optimum solution may exist for any given datacenter, with different size switches for the different switchingtechnologies (electronic, and optical time, wavelength and space).

For embodiments of the invention utilizing a plurality of opticalcircuit switch levels, these levels may be configured in a hierarchicalmanner—i.e., descending levels of granularity from the server level. Forexample, optical space/fiber switches select from a plurality of fibers,each fiber having multiple wavebands; each of said wavebands includes aplurality of wavelengths; each of said wavelengths are capable of havinga plurality of time slots. Note that said descending levels ofgranularity need not be consecutive—for example, data centerarchitecture 205, which does not include a waveband optical circuitswitch level between ToR space switch level 260 and WDM level 230. Inother embodiments, waveband switching may be used to instead of WDMlevel 230.

FIG. 3 illustrates two-level folded clos switching architecturesaccording to embodiments of the disclosure. Data center architecture 300illustrates a two-level packet switching architecture comprised of ToRelectronic packet switch level 302 and spine electronic packet switchlevel 304. This data center includes three electronic switch delays—fromthe ToR electronic packet switch level twice, and the spine electronicpacket switch level once.

Data center architecture 310 includes ToR electronic packet switch level312 and optical space/fiber switch level 314 serving as a spine level ofswitches; Data center architecture 320 includes optical ToR space/fiberswitch level 322 and spine electronic packet switch level 324. Thus, inthese embodiments, the amount of electronic switch delays are reduced;data center architecture 310 includes two electronic switch delays (ToRelectronic packet switch level 312 twice) while data center architecture320 includes one electronic switch delay (spine electronic packet switchlevel 324 once).

Data center architectures 310 and 320 each include at least oneelectronic packet switching layer; this enables the servers to exchangepacket data. However, in some embodiments, electronic packet switchlevels may be eliminated entirely. Data center architecture 330 includesoptical ToR space/fiber switch level 332 and WDM spine switch level 334.This data center architecture includes no electronic switch delays. Ifneeded, packet manipulations may be performed within the servers. Asdescribed above, for data center architectures including multipleoptical circuit switch levels, said levels must be configured indescending levels of granularity to the server level.

FIG. 4 illustrates three-level folded clos switching architecturesaccording to embodiments of the disclosure. Data center architecture 400illustrates three-level electronic packet switching architecturecomprised of ToR electronic packet switch level 402, aggregateelectronic packet switch level 404 and spine electronic packet switchlevel 406. This data center includes five electronic switch delays—fromboth the ToR electronic packet switch level and the aggregate electronicpacket switch level twice, and the spine electronic packet switch levelonce.

As described above, any of the electronic packet switch levels of datacenter architecture 400 may be replaced with an optical circuit switchlevel to reduce electronic switch delays. Embodiments not illustratedinclude data center architectures having a single optical switchinglayer. Two embodiments for data center architectures including twooptical switching layers are shown. Data center architecture 410 isshown to include electronic ToR electronic packet switch level 412,optical space/fiber switch level 414 and spine WDM switch level 416.Data center architecture 420 is shown to include ToR space/fiber switchlevel 422, WDM switch level 424, and spine electronic packet switchlevel 426. In these embodiments, the amount of electronic switch delaysare reduced—data center architecture 410 includes two electronic switchdelays (from ToR electronic packet switch level 412 twice) while datacenter architecture 420 includes one electronic switch delay (from spineelectronic packet switch level 424 once).

Another embodiment for a three level folded clos switching architectureincluding two optical switching layers—an aggregate electronic packetswitching layer between a ToR optical switching layer and a spineoptical switching layer, is not shown. Furthermore, data centerarchitecture 430 includes all optical switching layers, including ToRspace/fiber switch level 432, WDM switch level 434, and spine OTDMswitch level 436. If needed, packet manipulations may be performedwithin the servers.

Similar embodiments may exist for any n-level folded clos switchingarchitectures. For data center architectures including multiple opticalcircuit switch levels, said levels may comprise any type of opticalcircuit switch layer, but should be configured in descending levels ofgranularity to the server level.

FIG. 5A and FIG. 5B illustrates configurable bandwidths for switchingarchitectures according to embodiments of the disclosure. FIG. 5Aillustrates a two-level folded clos switching architecture with generalswitching sizes. In this embodiment, various network components andattributes for data center architecture 500 are defined as:

Number of spine switches=N

Number of ToR switches=L

Number of Servers per ToR=M

Number of servers=M×L

Bit rate per link=R

Bit rate per server=S; bit rate per link R×spine ports N/servers per ToRM

-   -   S=R×N/M

Number of servers per ToR=M=R×N/S

Bisectional Bandwidth=Z=Number of servers×Bit Rate per server

-   -   Z=M×L×S    -   or    -   Z=M×L×(R×N/M)=L×R×N    -   Z=Bit rate per link×number of ToR switches×number of spine        switches

Note that the above described bisectional bandwidth is the maximumpossible for folded Clos architecture 500. Two-level data centers may beselected in order to reduce the latency inherent in 3+ level datacenters for architectures that comprise only electronic packet switchinglayers. For these two-level data centers, increasing the bisectionalbandwidth requires the use of extremely large packet switches.

FIG. 5B illustrates data center architecture 500 in addition to modifieddata center architecture 510, which includes one optical switchinglayer, and modified data center architecture 520, which includes twooptical switching layers.

The bisectional bandwidth of data center architecture 510 is that of athree-level folded clos architecture (L×P×N×R), which the bisectionalbandwidth of data center architecture 520 is that of a four-level foldedclos architecture (L×P×Q×N×R); however, both data center architecturescomprise the same latency as data center architecture 500. Thus, thebisectional bandwidth of data center architecture 500 may be increasedwithout increasing latency or requiring larger packet switches.

Furthermore, the use of one or more optical switching layers betweenelectronic packet switching layers allows for theconfigurability/re-configurability of data center implementations. Forexample, in data center architecture 510, the illustrated WDM switchlevel may be adjusted to interconnect all of the servers individually,or may be used to set up multiple decoupled partitions of servers toprovide very-low-latency, high-performance server-to-server connectivity(i.e., one or more spine switches dedicated to interconnecting differentsubsets of the servers of the data center). Such partitioning may beuseful to data centers providing more complex, i.e., morecomputationally intensive, applications that require much moreinteractions among servers than to external clients.

Embodiments thus describe a hierarchical network of switches comprisinga plurality of top of rack (ToR) switches, arranged in a first level ofswitches, for communicatively coupling to a plurality of host devices ina network rack, and a plurality of spine switches, arranged in a secondlevel of switches, communicatively coupled to the plurality of ToRswitches to provide interconnecting communication links to the pluralityof host devices. In these embodiments, the hierarchical network ofswitches includes at least one level of optical circuit switches. Forexample, the hierarchical network of switches may comprise a folded-Closnetwork of switches.

In some embodiments, the at least one level of optical circuit switchescomprises at least one of: a plurality of optical fiber switches, aplurality of waveband switches, a plurality of wavelength switches, or aplurality of time domain switches. In some embodiments, the hierarchicalnetwork of switches consists of two levels of switches; in some of theseembodiments, the two levels of switches comprises a level of opticalcircuit switches and a level of electronic packet switches, or twolevels of switches comprises two levels of optical circuit switches.

In some embodiments, the hierarchical network of switches comprises morethan two levels of switches. In some of these embodiments, each of thelevel of switches comprises a level of optical circuit switches; inother embodiments, the hierarchical network of switches comprises atleast one level of electronic packet switches. For example, each of theplurality of ToR switches and the plurality of spine switches compriseelectronic packet switches, and the at least one level of opticalcircuit switches comprises a level of aggregation/distribution switchesto interconnect the plurality of ToR switches and the plurality of spineswitches. In other embodiments, the at least one level of plurality ofoptical circuit switches include one or more switches for upward trafficfrom the plurality of ToR switches to the plurality of spine switches,and an equal amount of switches for downward traffic from the pluralityof spine switches to the plurality of ToR switches. In some embodiments,the at least one level of optical circuit switches is configured toestablish multiple decoupled partitions of the plurality of hostdevices.

In some embodiments, the plurality of time domain switches comprises atleast one of one of a plurality of cyclic Optical Time DivisionMultiplex (OTDM) switches, or a plurality of configurable OTDM switches.In some embodiments, the plurality of wavelength switches comprises atleast one of a plurality of cyclic arrayed waveguide grating router(AWGRs) coupled to tunable lasers, one or more wavelength divisionmultiplexing (WDM) devices, or a plurality of wavelength selectiveswitches. In some embodiments, the plurality of optical fiber switchescomprises at least one of a plurality of fiber patch panels, or aplurality of micro-electrical mechanical systems (MEMS) space switches.

Embodiments describe a method comprising operations for, in ahierarchical network of electronic packet switches including a pluralityof top of rack (ToR) switches, arranged in a first level of level ofelectronic packet switches, for communicatively coupling to a pluralityof host devices in a network rack, and a plurality of spine switches,arranged in a second level of electronic packet switches,communicatively coupled to the plurality of ToR switches to provideinterconnecting communication links to the plurality of host devices,increasing a bisectional bandwidth of the hierarchical network ofelectronic packet switches by utilizing at least one level of opticalcircuit switches for interconnecting the first level of electronicpacket switches and the second level of electronic packet switches.

In some embodiments, the method further includes operations forestablishing multiple decoupled partitions of the plurality of hostdevices via the at least one level of optical circuit switches. In someembodiments, the hierarchical network of electronic packet switchescomprises a folded-Clos network of switches.

Reference throughout the foregoing specification to “one embodiment” or“an embodiment” means that a particular feature, structure orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, appearancesof the phrases “in one embodiment” or “in an embodiment” in variousplaces throughout the specification are not necessarily all referring tothe same embodiment. Furthermore, the particular features, structures orcharacteristics may be combined in any suitable manner in one or moreembodiments. In addition, it is appreciated that the figures providedare for explanation purposes to persons ordinarily skilled in the artand that the drawings are not necessarily drawn to scale. It is to beunderstood that the various regions, levels and structures of figuresmay vary in size and dimensions.

The above described embodiments of the invention may comprise SOI orsilicon based (e.g., silicon nitride (SiN)) devices, or may comprisedevices formed from both silicon and a non-silicon material. Saidnon-silicon material (alternatively referred to as “heterogeneousmaterial”) may comprise one of III-V material, magneto-optic material,or crystal substrate material.

III-V semiconductors have elements that are found in group III and groupV of the periodic table (e.g., Indium Gallium Arsenide Phosphide(InGaAsP), Gallium Indium Arsenide Nitride (GaInAsN)). The carrierdispersion effects of III-V based materials may be significantly higherthan in silicon based materials, as electron speed in III-Vsemiconductors is much faster than that in silicon. In addition, III-Vmaterials have a direct bandgap which enables efficient creation oflight from electrical pumping. Thus, III-V semiconductor materialsenable photonic operations with an increased efficiency over silicon forboth generating light and modulating the refractive index of light.

Thus, III-V semiconductor materials enable photonic operation with anincreased efficiency at generating light from electricity and convertinglight back into electricity. The low optical loss and high qualityoxides of silicon are thus combined with the electro-optic efficiency ofIII-V semiconductors in the heterogeneous optical devices describedbelow; in embodiments of the invention, said heterogeneous devicesutilize low loss heterogeneous optical waveguide transitions between thedevices' heterogeneous and silicon-only waveguides.

Magneto-optic materials allow heterogeneous PICs to operate based on themagneto-optic (MO) effect. Such devices may utilize the Faraday Effect,in which the magnetic field associated with an electrical signalmodulates an optical beam, offering high bandwidth modulation, androtates the electric field of the optical mode enabling opticalisolators. Said magneto-optic materials may comprise, for example,materials such as such as iron, cobalt, or yttrium iron garnet (YIG).

Crystal substrate materials provide heterogeneous PICs with a highelectro-mechanical coupling, linear electro optic coefficient, lowtransmission loss, and stable physical and chemical properties. Saidcrystal substrate materials may comprise, for example, lithium niobate(LiNbO3) or lithium tantalate (LiTaO3).

In the foregoing detailed description, the method and apparatus of thepresent invention have been described with reference to specificexemplary embodiments thereof. It will, however, be evident that variousmodifications and changes may be made thereto without departing from thebroader spirit and scope of the present invention. The presentspecification and figures are accordingly to be regarded as illustrativerather than restrictive.

The invention claimed is:
 1. A method for routing data in a switchnetwork, the method comprising: receiving, by the switch network,optical data from a plurality of node devices, the switch networkcomprising: a first level of optical switches and a second level ofoptical switches that are located between the first level of opticalswitches and the plurality of node devices, the second level of opticalswitches being configured to change pathways of light between the firstlevel of optical switches and the plurality of node devices; and a levelof electronic switches that are connected to the first level of opticalswitches, wherein the first level of optical switches are between theelectronic switches and the second level of optical switches in theswitch network; changing one or more optical pathways between one ormore optical switches of the first level of optical switches and theplurality of node devices using one or more optical switches of thesecond level of optical switches; and receiving additional optical databy the one or more optical switches of the first level using the changedone or more optical pathways.
 2. The method of claim 1, wherein thefirst level of optical switches comprise one or more wavelength divisionmultiplexing (WDM) switches.
 3. The method of claim 1, wherein thesecond level of optical switches comprise one or more optical timedivision multiplexing (OTDM) switches.
 4. The method of claim 1, whereinthe first level of optical switches comprise one or more wavelengthdivision multiplexing (WDM) switches, and wherein the switch networkcomprises a third level of optical switches that comprise one or moreoptical time division multiplexing (OTDM) switches, the WDM switchesbeing located between the OTDM switches and the second level of opticalswitches.
 5. The method of claim 1, wherein the plurality of nodedevices are included in a data center.
 6. The method of claim 5, whereinthe optical data from the plurality of node devices is addressed to oneor more external devices that are external to the data center.
 7. Themethod of claim 1, wherein the electronic switches are spine switches ofthe optical switch network.
 8. The method of claim 7, wherein the switchnetwork is a folded clos switch network.
 9. The method of claim 1,wherein the second level of optical switches are communicatively coupledto the plurality of node devices using optical fibers.
 10. The method ofclaim 1, wherein the plurality of node devices comprises a plurality ofservers.
 11. The method of claim 1, wherein the plurality of nodedevices comprises a plurality of storage devices.
 12. The method ofclaim 1, wherein the second level of optical switches comprise one ormore fiber patch panels.
 13. The method of claim 1, wherein the secondlevel of optical switches are micro-electrical mechanical systemswitches.
 14. An optical switch arrangement, comprising: a first levelof optical switches operatively coupled to communicate data from aplurality of node devices; a second level of optical switches coupled tothe first level of optical switches and operatively coupled to theplurality of node devices, the second level of optical switches beinglocated between the first level of optical switches and the node devicesand configured to change one or more optical pathways of lightpropagating between the first level of optical switches and theplurality of node devices; and a level of electronic switches that areconnected to the first level of optical switches, wherein the firstlevel of optical switches are between the electronic switches and thesecond level of optical switches.
 15. The optical switch arrangement ofclaim 14, wherein the first level of optical switches comprise one ormore wavelength division multiplexing (WDM) switches.
 16. The opticalswitch arrangement of claim 14, wherein the first level of opticalswitches comprise one or more optical time division multiplexing (OTDM)switches.
 17. The optical switch arrangement of claim 14, wherein thefirst level of optical switches include one or more wavelength divisionmultiplexing (WDM) switches, and wherein the optical switch arrangementcomprises a third level of optical switches that comprise one or moreoptical time division multiplexing (OTDM) switches, the WDM switchesbeing located between the OTDM switches and the second level opticalswitches.
 18. The optical switch arrangement of claim 14, wherein theplurality of node devices comprises a plurality of servers.
 19. Theoptical switch arrangement of claim 14, wherein the second level ofoptical switches comprise at least one or more of: fiber patch panel, amicro-electrical mechanical system switch.
 20. The optical switcharrangement of claim 14, wherein the electronic switches are spineswitches.